invalid report, what next?

You can report them to Valve.
In order to do that, you have to:
1. Visit the accuseds profile
2. Click MORE drop-down located at the top right of the page
3. Choose REPORT VIOLATION
4. Select the violation, then describe it and provide the evidence
5. Click SUBMIT REPORT

Click here to view an animated gif that shows how to report a violation.
But do keep in mind
https://forums.steamrep.com/threads/general-discussion-faq.132723/
#3

Hidden Content:

**Hidden Content: Content of this hidden block can only be seen by members of (usergroups: Administrative, Moderating).**

Generally speaking, we need the following documentation: (written guidelines, for reference)

1. Proof of a specific agreement. This includes what was to be traded for what.
2. Proof of transaction and delivery: Proof that you upheld your side of the bargain, per trade agreement in #1, as well as proof that the accused did not.
3. Proof of identity; all of the above needs to be verifiably tied to a Steam account, and we need to be able to confirm you really did interact with the specific accused you named and not an impostor or something.

If the specific terms of your trade weren't negotiated through Steam, it's usually difficult or impossible to establish #1, so your report may have been closed for that.

I'm sorry to say, you're in the wrong place for that. We aren't in the business of "avenging" victims, or striking fear into the hearts of scammers. The purpose of reporting someone is to warn others not to deal with him/her. If you have solid enough evidence, we'll permanently add it to their SteamRep profile, so others can really quickly and easily find it before they too find themselves scam victims.

Hidden Content:

**Hidden Content: Content of this hidden block can only be seen by members of (usergroups: Administrative, Moderating).**

Unfortunately, if you don't have a trade agreement or other evidence criteria, we can't publicly call them a de facto scammer. We try to hold a high standard of evidence before trashing people's reputation.

You can try reporting to other subreddits to ban him if they feel your evidence is solid enough, or reporting to Valve as Horse suggested, but it seems like you weren't able to provide enough to meet our standards. Sorry about that.