Stop letting bot traffic evict your customers' sessions. A simple Redis configuration switch from LRU to LFU solved our crawler problem, with a Django configuration example.