When sea level rises, does the increased ocean area cause the earth to cool down somewhat?
As you say, land and water albedos are very different. Meaning that a square meter of water, in average, absorbs more energy from the sun than a square meter of land. Also water transfers heat vertically more efficiently than land due to convection and it also affects climate by providing a ready moisture source.

That said, changes in the surface of land vs water do have an impact on climate. However, the change in water surface area due to sea level rise is very small compared with other changes in land/water surfaces. Let's crunch some numbers to see it more clearly...

Lets consider a good approximate of the length of the global shoreline of 620,000 km (a precise number is impossible to get due to the coastline paradox). Now, at current sea level rise of 3.2 mm/year, 30 years would mean about 10 cm of sea level rise. If we assume that the typical slope is such that this moves the shoreline also 10 cm forward, the associated area increase would be:

$620000\, \text{km} \times 0.0001\, \text{km} = 62\, \text{km}^2$

And this is negligible compared with other effects like the drying of the Aral sea (~28,000 km$^2$), the construction of the Three Gorges Dam (1,084 km$^2$), or even the construction of the Palm Islands in Dubai (80 km$^2$).

Even if our very rough calculation above is multiplied by ten or even one hundred, the contribution of sea level rise would remain small compared with other changes.

At a global scale over the last 30 years there was a net increase of 58,000 km$^2$ of water surface, but mostly due to natural processes or large infrastructure projects. Sea level rise played just a very minor role in those changes.  Indeed the following figure from the paper Earth's surface water change over the past 30 years shows changes from land to water in blue and changes from water to land in green:



Now, what's the effect of this increased water surface area (even if it is not due to sea level rise), that's another question that I won't get into in detail. But in general it will be a warming effect, both because of the lower albedo of water, and because of the warming effect of low clouds. However, the actual effect will depend on where the new water bodies are located; some extra water near the sea wouldn't have the same impact as some extra water in the middle of a desert.