Nerve-racking is the more common spelling, but nerve-wracking is also a real and accepted variant in many dictionaries and style references. Both forms mean extremely stressful or very anxiety inducing.
The more common version is nerve-racking, likely because it connects to the idea of something that racks the nerves, meaning it strains or troubles them. Even so, nerve-wracking appears often enough in edited writing that it should not be treated as an error automatically.
- More common: It was a nerve-racking interview.
- Accepted variant: Waiting for the results was nerve-wracking.
If you are writing for school, work, or publication, it is usually safest to choose nerve-racking because it is the standard form most readers expect. If a house style or dictionary you follow accepts nerve-wracking, that spelling can also be fine. The key is to stay consistent within the same piece of writing.