Custom missions can be made in two ways. The first way is to use the "select airplane" command in the Ysflight menu right after making a flight in the game. This will prompt you to select an aircraft, then a starting position. The result of this will be that you will fly your chosen aircraft in the same flight file you just flew before. This means, for example, you can do an aerobatic demonstration with a single ship, exit the flight, select another aircraft and follow the first flight you did as a Wingman, and thus build up a demo team. The uses are just sort of endless really. People have made "air traffic files" which they save as a .yfs (this is done through file->save flight) file, and can then run by default on that map by adding it to the .lst file line that map is in. Others have made really excellent movies by continually adding more stuff to a flight, until an incredible scene results. Others still have made incredibly fun combat scenarios that are sometimes much more challenging than what you could otherwise encounter, even in the dogfight generator.
The second way custom missions are made is by coding the mission. This is done by opening the .yfs file in a text editor like notepad (or making one from scratch) and adding lines from the mission syntax to make the game simulate things. What is more commonly done is taking an existing .yfs file created through the aforementionned process. This is one of the great unexplored features of ysflight, since you can make the game do literally anything with this process. The japanese modders at 2ch have already deciphered the functions in the mission syntax, and a file can be found at this page.