One of the first things I learned when I began to edit was that I needed a copy of the Chicago Manual of Style. Over the years, it's been an invaluable tool for me, for everything common and uncommon while working on manuscripts. There's a reason why it's known as the "editor's bible."
After I got over my initial "how in the WORLD do I find my way around this thing?" phase, I came to love this thick beauty of a reference manual. If you don't know basic grammar rules, this is the book for you. If you know basic grammar rules but can't remember whether your numbers should be spelled out or written as numerals, this is the book for you. Have you forgotten when to capitalize titles and honorifics and when not to? Well, this is . . . yes, the book for you.
The Chicago Manual is THE go-to when editing works of fiction and more. Though some online resources such as Grammar Girl talk about CMS as if it were one of many options for fiction writing, the fact is that it's used as the standard by most publishers as a formatting guide, so it's probably best that you get used to it—especially if you're submitting your work to a publishing house.
On the other hand, The Associated Press Stylebook and Libel Manual is used for magazines, newspapers, and similar publications. The differences between the AP Stylebook and Chicago are sometimes subtle, sometimes obvious—and in some cases, the rules are the same.
The key to using these guides is consistency, no matter what you're doing. If you've been using the wrong one for the wrong purpose, then at least be consistent in your usage. There are many times that AP will put quotation marks around something that Chicago puts in italics. As long as you're not using both styles in one book, you're not as bad off as you might have been.
Still, it's best to keep some general guidelines in mind when trying to figure out what to do in any given editing situation, especially if you tend to write magazine articles as well as full-length novels. The most general, sweeping rule is this: AP style is geared toward brevity and space-saving. If you keep that in mind, most of the differing rules will be easy to remember and will make sense. Newspapers and magazines are all about cramming the most information in the smallest space possible, so this is key to any memory tricks you'll want to employ.
For example, AP does not use the Oxford comma. Yes, I know, this is heresy. I swoon at the mere thought of not having that comma, and consider anyone who doesn't use it to be a moron of the highest (or lowest?) degree. I'm sure we've all seen the cartoon example of why the Oxford comma is such a necessity:
I can't tell you how important that comma is . . . I don't want to see any stripper who looks like Stalin or JFK. Uh . . . or any other stripper, but that's not the point I'm trying to make here. Just use it, okay?
Numbers and numerals differ with the two style guides. AP is all about saving a few character spaces, so they recommend the use of numerals in almost every case. Chicago states that numbers should be written out up to ninety-nine, and then numerals after that—though it does acknowledge an "alternate" rule of writing them out only up to ten if you prefer.
Also differing between AP and Chicago is the use of spaces. Again, AP typically wants brevity. When using ellipses, a newspaper will place them ... like so. Space, three dots, space. However, in novels or other books, you will see them placed . . . in this way. Space dot space dot space dot space. In the various books I've edited, I've frequently come across... this, where there is no space prior to the ellipsis but a space after it. Seriously, I have no clue where that one comes from, and yet I see it from a variety of people. I did have one author tell me (after I did an eval for her) that another editor had told her this was the way to do it for "proper" formatting for e-books and that print books were "different." I've never heard of that and can't imagine why it's thought to be so. It's wrong, plain and simple.
A seemingly contrary use of space comes with the em dash (one of my favorite things to insert in a sentence). The AP Stylebook recommends a space before and after the em dash — like so, and Chicago recommends no space, just butting that dash up against the words on either side—like this. Why does AP want to add space for this when they recommend space conservation everywhere else? No idea. It just is. And again, consistency within your manuscript is the key here. [Unrelated side note: I think the biggest problem I find with em dashes, really, is that people seem to feel that a hyphen is the same thing. A hyphen, short and sweet, is used to connect words such as "quick-witted." An en dash (longer than a hyphen, shorter than an em dash) is used to show a range, such as "anywhere from 100–150 people attended."]
These are but a few of the most basic differences in the style guides, and after a while, you'll probably have them so ingrained that you'll begin to spot them in other people's work if done improperly.