After details analysis we are observing that at the time of issue in varnish, Server processes got increased and varnish giving incomplete request and returned to 504 to google load balancer. iam sharing below Google LB error and SAR command output --
{ "insertId": "1l6m956f37u7rz", "jsonPayload": { "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry", "backendTargetProjectNumber": "projects/488", "remoteIp": "106.197.5.134", "statusDetails": "client_disconnected_before_any_response", "cacheDecision": [ "CACHE_MODE_USE_ORIGIN_HEADERS" ] }, "httpRequest": { "requestMethod": "POST", "requestUrl": "https://abc/iry", "requestSize": "364", "userAgent": "Mozilla/5.0 (iPad; CPU OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/125.0.6422.80 Mobile/15E148 Safari/604.1", "remoteIp": "106.197.5.134", "referer": "https://xyz/efg/w34pro-smartwatch-23944871197.html?pos=2&kwd=smart%20watch&tags=A|PL|||8752.144|Price|product|||LSlc|rsf:pl-|-res:RC4|ktp:N0|stype:attr=1|mtp:G|grpfl:45|wc:2|qr_nm:gd|com-cf:nl|ptrs:na|mc:184363|cat:248|qry_typ:P|lang:en|flavl:10|cs:9555", "latency": "0.024914s" }, "resource": { "type": "http_load_balancer", "labels": { "zone": "global", "forwarding_rule_name": "-logical-seperation-443lb", "target_proxy_name": "logical-speration-lb-target-proxy-2", "backend_service_name": "varnish-group3", "url_map_name": "logical-speration-lb", "project_id": "abc" } }, "timestamp": "2025-01-08T03:32:37.468854Z", "severity": "INFO", "logName": "projects/987/logs/requests",
} ------------output of SAR command --when the process increases from 1500 to 3k and 4k issue error started, coming--
03:00:07 IST 4 1543 1.42 1.52 1.55 1 03:01:07 IST 2 1547 1.31 1.48 1.53 1
03:01:07 IST runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 03:02:06 IST 4 2044 1.65 1.55 1.55 0 03:03:06 IST 1 4079 1.38 1.49 1.53 0 03:04:06 IST 1 4224 1.67 1.54 1.55 0 03:05:06 IST 2 4228 1.69 1.58 1.56 1 03:06:06 IST 1 4223 1.43 1.53 1.54 2 03:07:06 IST 1 4208 1.60 1.57 1.56 0 03:08:06 IST 1 4196 1.54 1.54 1.55 0 03:09:06 IST 1 4063 1.66 1.58 1.56 0 03:10:06 IST 1 3822 1.58 1.58 1.56 0 03:11:06 IST 1 3592 1.56 1.55 1.55 0 03:12:06 IST 2 3349 1.24 1.46 1.52 0 03:13:06 IST 1 3098 1.29 1.44 1.50 0 03:14:06 IST 1 2863 1.41 1.46 1.51 0 03:15:06 IST 1 2618 1.36 1.43 1.50 0 03:16:06 IST 1 2391 1.85 1.57 1.54 0 03:17:06 IST 2 2147 1.52 1.53 1.53 0
==Sharing log below == 20241218211542 - - - 0 - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.108 Mobile Safari/537.36 (compatible; Googlebot/2.1; )