IISER Kolkata is the best IISER for Physics. It is especially good for Astrophysics. The ongoing research in the area is top-notch. There are labs such as CESSI (Center of Excellence in Space Sciences, India) inside the campus, spearheading research in Astrophysics and Astronomy area.
The faculties are well experienced and put more emphasis on building the fundamental concepts on priority.
One of my friends is pursuing his UG from IISER Pune and according to him, it is a great option if you are interested in doing a course in Physics. Modern science is no longer segregated into strict boundaries like Physics, Chemistry, Biology. You will need to learn everything a bit if you want to have a career in exciting interdisciplinary stuff. What makes IISERs best is that students get to learn Physics, Chemistry, Biology, Mathematics, and other IDCs for two years.
Considering all these factors, I will say IISER Pune is the best option if you are interested in Physics.