I like the Teensy, and it is like arduino on steroids, on speed. There is a slightly older, cheaper and lower power consumption Teensy 3.2 still in production. It (also) has a built-in DSP you could use for detection algorithms?
PMW3360 is a great high resolution mice sensor (very fast response, low power management, several resolution configuration register...). But it's been designed to focus on small elements (like dust) using a recommended lens (LM19).
Identify important challenges:
Any idea on how to use it to detect bigger elements like ants? There is probably a several ways to do it, but the sensor is designed to focus on very tiny particles. So I was wondering if we should use another lens or try some other sensor specification from the datasheet...
What kind of microcontroler are you planning to use? Simple Arduino AVR like the one on the Instructables fiber optic link don't have much low power management and are very low speed.
Inspire novel solutions & Challenge assumptions of the problem:
I used to play with "low cost" mice sensors like ADNS-3050. They have an interesting register call "Pixel_grab" where you can read the image to make you own detection algorithm. If the PMW3360 has hard time seing and ant, you can probably use those ADNS-3050 with a Teensy board (Teensy3.6 has an amazing speed to read the 361 pixels). The ADNS-3050 cost only few dollars on Ebay and also have great low power management.
It you can create your own detection algorithm, you will be more flexible on the type of insect you want to detect.
Teensy 3.2 is also a great board. Cortex ARM 32 bits architecture change everything compare to the small Arduino UNO... And I am not talking about clock speed, it's insane!
For low power project I usually go with STM32L controller. They is so many that you can find the exact one for your need and the low power modes are fantastic. The NUCLEO version cost nothing if you want a new toy :p.
But it's not Arduino compatible, so way harder C embedded development :(
yeah i have used a bunch of the mice sensors, one of the trickiest things is making my own lenses for them. The PMW3360 has lots of nice features in librarys, but the pmw3901 is actually setup to do optical flow, and a lens more appropriate for what i want, but its pixel grab functions are locked off behind a proprietary thing