Make sure the pressure you apply is light. If you put too much weight on the file, you can cause it to become clogged with the material, break some of the teeth (sometimes called ‘shelling’) or cause the file to create harmful vibrations in the material (referred to as ‘chattering’).
You should also use your non-dominant hand to guide the file to the right as you make a stroke. The trick is to still keep the file at right angles to the workpiece as you do so!