Two components comprise tabletop role-playing games (RPG): the game system -- the mechanics by which success or failure are determined -- and setting, which is the world in which the game takes place. These two elements are commonly referred to as "crunch" and "fluff" respectively. To create your own role-playing game, you need to determine how you want the system to work and what the world is going to be like. If you have no interest in creating a new game mechanic, you may want to use an existing system and tailor it to your setting of choice. This is called "reskinning."
Select a randomizer. Using a randomizer creates dramatic tension in a game. When there always exists a chance of improbable victory or unforeseen defeat, players can never know exactly how their actions will play out. Most RPGs use dice. Other options include coins, cards or roshambo. "Dread" uses a "Jenga" tower. Not every game system uses a randomizer, however. If you feel that a random element doesn't suit your game, don't use one.
Choose a mechanic to determine success. With dice, there are three basic types of mechanic: fixed-die, dice pools and step-die. Fixed-die systems use a specific number and type of dice for every roll, regardless of a player's stats. Dice pools use a number of dice based on a player's stats. Step-die mechanics change the type of die used for each roll. The D20 system and GURPS are fixed-die systems, while Whitewolf's Storyteller and the One-Roll Engine are dice pool systems. Savage Worlds is a step-die mechanic wherein your abilities are rated by die type (d4, d6, d8, etc). If you are using a different randomizer, you'll need to come up with a way to tell if players have succeeded or failed.
Determine the desired probability for success. In a linear dice mechanic, like the D20 system, the probability of rolling any number, one through 20, is exactly the same (5 percent). A 3d6 system creates a bell curve, where the probability of rolling a mid-range number is much higher than rolling a three or an 18.
Determine whether you want to use modifiers or not. Most game systems use some type of modifier, based on a character's skills, to add to, subtract from or set the target number. This allows a character to become better at a certain skill while still using the same number and type of dice for every test. In GURPS, for example, a character's skill is the target number and the goal is to roll under it. In D20, a character's ability or skill correlates to a modifier that is added to the dice roll to make it easier to match or exceed the target number.
Create a magic system. This may not be necessary depending on the type of game you are creating, but many games use some type of spells or powers. Magic systems often have a power source (mana, stamina, power) that determines how many spells or powers a character can use before resting. This prevents magic from overpowering more mundane skills and abilities. If you use a depleting power source, determine how much each spell "costs" to cast, how power is regenerated and whether there are ways to store power and use it when you need it (power stones, spell batteries, magic rings).
Design a character creation system. This is the system that your players will use to create their characters. Most systems use some basic ability scores (strength, dexterity, constitution, intelligence, wisdom, charisma) that determine the basic attributes of the characters. In addition to these, many systems add skills. These allow characters to specialize and focus their talents. Skills can be as general or specific as you like. For example, you may have a "manual dexterity" skill -- or break that down into more specific aspects like "lock-picking," "sleight-of-hand" or "pickpocketing."
Determine how characters gain skills and abilities. Some games use dice to randomly generate scores that are assigned to abilities. Others use a point-buy system wherein each character has the same number of points to distribute as they like.
Determine how characters advance. Most games include a way for characters to develop and grow by becoming better at what they can do or learning new things. Two ways of doing this include experience points and character points. Players gain experience points when they complete a quest, kill a monster or solve a puzzle. These points are added together until a certain threshold is reached, at which time the character gains a predetermined number of ability or skill points, powers or spells. Character points translate directly into skill points and can be spent as they are gained or whenever the player wishes.
Choose a genre. Fantasy and science fiction are two of the most common genres in role-playing games, but they aren't the only options. Pulp, horror, whodunnit and comedy are also good choices.
Create a setting. This can be an entirely new world or a version of a pre-existing one. A fantasy game, for example, may be set in a vaguely medieval Europe (like "Dungeons and Dragons") or Republican Rome, Ancient China, Egypt under the Pharaohs or any place and time that interests you.
Design your monsters and villains. These should be specific to your setting. You are unlikely to run into robots in Regency England or a griffin in modern-day America. That said, if you have good, story-driven reasons to include those elements, feel free. You can create a set of common enemies that you want to inhabit your world or create a system by which you can generate new threats as needed.
Select an open-source gaming system. Some examples include the One-Roll Engine, FATE, Savage Worlds and the D20 system.
Choose a setting. You can create a game based on a pre-existing setting or design your own. If you plan to publish your game, choose a setting that is either in the public domain (such as "Alice in Wonderland" or "Don Quixote") or an original idea.
Decide which elements of your setting relate to each attribute, skill, advantage, disadvantage or power. These elements will vary greatly depending on the system and the setting you've chosen. This may involve creating new powers, but doesn't have to. In an "Alice in Wonderland" game, for example, characters may have a power that allows them to shrink or grow at will. An example of a disadvantage in this setting would be "curious" while an advantage would be "brave." To avoid game imbalance, use existing powers as a template.
Create monsters and enemies. These are the standard villains your players will encounter. For example, an "Alice in Wonderland" game would have card soldiers as a common monster. Again, using existing monsters or bad guys and changing the names of their powers will help you avoid imbalance.