sentry中的worker错误解决[ERROR] sentry.errors.events: preprocess.failed.empty

sentry错误解决 worker failing with [ERROR] sentry.errors.events: preprocess.failed.empty

sentry 的worker 发生如下错误：

11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c0b43b415b254dcabf25b4a59e8bec98:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0cfcb7ece84d49ea83f4fb555781c1ed:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:133eede1c2714bda9f07057eea1c11ae:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:250141f3573a46d28cacf83c9bf05183:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:1ec1a61183104cccba9d398c6418acc5:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:3fea2f2c12654e67aada95ce52248502:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:db5499194b5f4f3baa4b6e21f0ebe044:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:4cf2e27681fc462394981237168a88e1:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:dce53fb9f482450ab550bceed6e59f86:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:678df00fafbe43bf992c8be50a07b311:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b7fb37edadf141af8752872dae8d746d:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:88e5c19a24a547eeb14f2383032ac6fc:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:5ca86fd53b5f4015aa54ac4bc1be5f95:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:4237b0474a9e45aa87de0de682d1eb0a:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:23c44ce06b864a2a9e5091fd63db2dff:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:16d63999c36e45d2a7badc0cd6d4b2f9:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:3fb0065b10ed4030bdb928f17d641fd8:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b0e6ef751c7a42a78e63c253f0fe77ad:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:84a7432291614426bc281e93f00d7cfa:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:82b66d5752e1473ab778fd0560badb00:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0f6967ebca624232a825af4da5fc1cbc:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:d50e0c09d61c4d8f8e21344590d88846:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:18cb711c32944877954ae175269cbc22:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c6b280cfc77646c28d568f9ad1a167a3:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b888421107f44b45be654142bf9bb3c1:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:318832bcc08040e1820307b91eb4df1f:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:12c41a96d8604aa2ac27c5d102695940:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8ed593cc0f39422bb7b44c622311fd80:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:58173f4944244bd9b2cb2a487b2f224b:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:bf42f907bf974c17aafac17236bc00fc:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:43c81367b4c248bdb5f33a8ceb2bcdee:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0d87ae88efff484b98af3ba40ea24e61:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0fe413fb8cc745178d633b5e59ac075a:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:f5edcd3c866e4cf98a11beae03c6d515:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e0ee92403480427481b576501e8a8515:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0466486290514c45bdcbde4fd645e103:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:09143efe7ee941fa911459407094a9a0:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:f0ca5dbb43834d0eafb0322fa041c8a5:24')
11:54:59 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e695d915464c47ca9e7d343e46136c28:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:3318a985a9044e75be6d8dce3d04565c:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:eb82a4bdd07d4f05803aed4f7fe6dc48:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:59b8294ff2cf4fa3b8aa2deea4ed3c05:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:feeb01576a314383a6d1e95aaf9b6f49:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:9a4bf4962b5c44de9d37cb11172a019a:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:6e0cfcd6f22844e092202b5b5c48103b:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c4897b4ae29c4c0489f6bfcbfb208084:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:245c26d3ed7c41bb8546ce18a8ec1f94:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b650061574b54cfb9877ef0f3435a2b4:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:be226b1f7c1e4a4b9fef6b7c4e4d631a:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b1f137051aee488bba18347611c669c6:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0e87723f28fc4899a61e1fdc511cc438:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0b9bbd460c654a6da2b40ba17c8e6156:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:01cf0ab8ca5d43318e8f8e99a149fceb:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:90557d321cfd4dcf897018deaa18d5ec:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:64ee0a84b1534816812f4a67d49232e5:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:795f01df6cfc43cba09f607836846fdf:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:61a731e0cf18473fa7348cbfc32110c9:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:942f1454914a4253ab8281e5e3f23467:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:321b72af14e841e4a74b9874efa7158e:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:11b937473e2642bcac3186b8dd4fc5f7:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:fc0177a073b4437f8db2e28f15b9a149:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:5d8defa9beb747948c59ecacd1c8159b:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:d24826178a374058862d2033332e4ee3:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:5c5755c296d744cab0c1638b1c9b8da2:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:59498dc642734a85898788cf9b1b4146:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0537257bfe51449ca9a71edcfd17ec23:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:1465fd576baf4bbc9aa7cc5207c6112d:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:fdcb3f896f0746f2bc0bf6745398cfe3:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:3e886b6b79db441dbc5bad1c3eb6f7e1:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:230d76db26a8420bb86d1e282513e0af:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0ab4e1c2c2d648368fc422336a5f2478:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:83dbf1130a15434d9abe0be0105b52b3:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:6a79915bfb7543138f140e7194988bbe:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:3d1b82d148cb454dbbf5941f92fec775:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:81be5c51effb4a6a85bbc9791850609a:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:92aa411d7b464d158b526cac51707a0e:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0cfd222bb3974d528b18e02714fe7c0c:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c0899cec34754184ae312f07f5820920:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:926aac4b03ef432ab901623ff06c11e6:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b1666ac85079452991851f547d80cfcc:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:9514260736384061b65ba6df1df48973:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:95006df32b5c4f5d9877e000e32df542:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:5cd283b4009e4058a7e7dedb416dfa6b:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:9a0448a5165f4b1ca52034df90f49541:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:14d04d8a5b9e42ed93e6753aa73b8d91:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:7750c16f0c8f49acb390cdb31362b477:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:53ce18e8b4124e7a9214bbea94d0dd61:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:9ef0be3ea644411ab06e822cb97fe4d9:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:a07107ad5ba94a96bfccc04f2a2e208b:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:43574c68a32445488e796c17c09212de:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0b1e5da3d1f946ed857ad082e43342b5:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e80917f3d7df4faf8ec1bee56e7eca73:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c8dfb660876a49248a9fcb107741ad30:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:2507b8f88a1a4671a803a2f124200ca7:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:7cd17ce060214bcc9c2387f92fe19422:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:340a0b98cadb4be29abd83a8d1961c4c:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8760b5747ba44aad80dadfd5b2897acf:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0e59929be88b48f8a1d2c7726186c8ad:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:dd2c7647d9e942da948a2230e953441d:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:dacea298849c4c8aad07ada874a6d1f9:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:244fbff060de421b925df6e0726d976a:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:27568b9fda024ea388ddd030b6ae9a02:24')
11:55:00 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:d8f1a02f4de24c95bd71416745ccdeb2:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:7ab6fc3ae4e4487aba32a310999d2053:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:acec8c4b910546a0b466572c43148396:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:5f990e88445543dcad076dbf79f551a3:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:425fab82640944cb964a206bd59c5339:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:adc0caf4db954a628c91ec744cd0cf7a:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:19d798e8041d4842b0c8826588b36267:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e4006ef2cb354abe80106cfab62189e8:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:92e413c8438249f9a890046993b8fb68:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:93d94e2998de45d3a594c6a6aee3a45d:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8c65644ffb6f43378819c8961efa4f78:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:85d0346dc59c4cf88a03ec84ec8b73c4:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:175c7b39144f420d8ebbbda33e4b46dc:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:7bc3f13197f14467925ad28fdf14ae3e:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:30fe26fc86bd4dad81321185d52f6ea8:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:4b32936621044704862ca9e5331b86a0:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b58893e2feef4f66b89fb11f36872a6d:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:a168eeb120a6409ea669227407f0691c:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:1130700e36684d60a21b497f1a96239d:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:7e79a027a357483a86c4143b75b65f6d:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:a159e0263da14f6982cf7f5894829fcb:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:060e02c484a644a8ad7c3ffceb4845b3:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:27222c35f1be4540832d99015dfc9463:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:056124bda56a462dadef4b3669c1cb0c:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:043e04d6400940fd9d6b523ae6499e4c:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:78426e30050c4ebfa7a64644f0cf4d47:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:eff151cf30644c0290c66d773c44c193:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:4f79b2515bc34a0a9b8bf61e4b10f090:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:70d826a947084fe1a1988318f25645bc:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:30f49f70c8314522be398c5ec2f625f6:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:1058fd4f50514fc9b28f1d367eec909f:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8c88dbc4005a44bab5a5cae4d6c02be8:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:2168c0d7e3c740b5b59c83df23ff3094:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:d6ed39069f634a35b2b042a9096f6346:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:7779d604c56643fd873e82d60fa3d3ca:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:f2590b91381d4fe68ae53c3b111bbb79:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b466377ad5e940c286f64efc65ce17f7:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0d08e31c669840978b372f885474cf61:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:7e69bb0694f740078bfb594345beac02:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8b142caca5884ad6994a8226b6230aa3:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e75e71e963464f52a4bd3f26c3073f9d:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:873da561737049e2a77f06553f144ae3:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:ed98d0138bc44937b5c0c4e59075b346:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e1b49e448fb34d8ab9c148ecedd6ce74:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0431e73bd55443579da46f1daf710013:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:cf888e021a42460390390ae6d371157b:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8547d13989914ac4b4a7a8876ae58677:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:6a0e4b2c5be0446ea2dd6b13abb1ef83:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:5d1ef3eb454c485a9f42a971495cf85e:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:a38b2f3ef114492e88dcccb4f3341997:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:1e5862abb70a41f59a40645dcd806a93:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e515f54e28ca44249fbe9e1f1d447c2b:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:63da481e9f124db7abcd5dede134aa54:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:d4982508f0894185ae930e6f286df3df:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:1a1e3944dd894c9c97e4f21179a590fd:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8aa7fb41438b4b14a45228650023d778:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:18975b2567aa493f95d9765513ff2e51:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:01d09a07224f4ae5b0245f19a660787d:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:abf7cca021e94c0c9cc65f9dd227bd5e:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:bf29b1a019c345a48a89c6099a8a5c94:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c87c0e4c2781410cabb46fa977e4cb65:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c676d3e852ac437b995a34d5b4546263:24')
11:55:01 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:a97777fa391a4d15be0818fa6810766a:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:575cdc31dcfb4d6786c55f895d61b543:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:47b772c837464b88b4834b14a982d43f:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:b0e8d0ba49ba4445b83c9036640eda2a:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:3184e08386d14cf8a5fcb16e0fdbe9a2:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:f5c01587663e4bc497e32cdde34a95a1:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:9684edd7d72f44e7b019a0ee353b9508:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:95440acbad12497caa15106112137e46:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:feb90e36c30f43f6bd05f6bdb13f94b6:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e8e8d511771049a39c3cab65048766d0:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:a5250601e902405b94ca5b7fef7a36f4:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:31112125875f43088ac5f5613b02965b:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:0ea9ab2768aa48d3b271ccb571f503f2:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c2de225ba295464bbcefc377c8f39bdd:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:a02e506cb8c247ffbf79982ffa1bd4f9:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:6d510beea55f419a975606eaa058f12b:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:96e7609ceb5542aea9979a5994daba76:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:6d3ba88400c643e8b19a394824014910:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:508e3f7f2a954998b4a555b9841756be:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:e2a8f90e57a14d08b390e583e2903b0a:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:2d136c1d4d0c4f3693b67be57d755c68:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:581f422268684044b6520dcfc0d67314:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c42ee7b7f9064d6ca63e8c25c249ba57:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:8f11c892c9d6410397c01ec9549aad56:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:491fd2ae77bc4d179042ca6aaa847fd9:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:68a85014c1bb42228803c99b1ce576ab:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:11c36d7e7f7f46ddb40336c80b1a3594:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:c470b735ea52453d81c9144ee60d4632:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:73172588ecb84520a1842f505ea2750a:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:47ed90bca0c14f5aa8eb3bb95967990e:24')
11:55:02 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:05e68335dc254783b874764567c3894a:24')
11:55:03 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=u'e:48ffe5ea1412456e881fc6553d2cad8e:24')

This error explicitly means “we can’t find the event in cache, but the job exists in the queue.”

This can happen in a couple scenarios:

These jobs were taking longer than 1 hour to get pulled from the queue and they got TTLd away, which is expected behavior.
The cache was purged and these keys are literally just missing.
Some other misconfiguration where maybe the blobs are being written into one cache, then trying to be read from a different one.
Other than that, I’m unsure what could be wrong. Hopefully this points you in the right direction.

Thanks for the info, it seems like the main issue was our server was being overwhelmed by reports causing it to fall back, and many of them started to timeout, I added a quata, and it seems like thing are back on track

I was wondering is there a document/article/diagram that shows the lifetime of a report as far as different system components are concerned? I’ve tried looking for one, but I wasn’t able to find it (besides https://docs.sentry.io/server/queue/ 225 which is pretty light on details)

Here is what I’ve gathered and guessed,:

Report is sent by client to dashboard node
Dashboard node stores the raw event in the cache you mentioned. (Is this cache the same as what is referred to as queue/broker)?
The report is picked up from the cache by the worker node and updated in Postgres. (What I’m confused about is why the worker can pick up the cache key but not the item? are they stored in different systems and only one of those systems has TTL?).

For what it’s worth, we ran into the same issue and ended up cleaning the backlog with:

sentry queues purge events.preprocess_event

From one of the worker nodes.

Once that was done, all backed up events were lost (as expected) but processing resumed as normal and new events are treated properly.