What is the meaning of the square brackets in ant colony optimization?
The square brackets $[]$ in $[\tau_{ij}]^\alpha$ and $[\eta_{ij}]^\beta$ may be just a way of emphasing that the elements $\tau_{ij} \in \mathbb{R}$ and $\eta_{ij} \in \mathbb{R}$ of respectively the matrices $\mathbf{\tau} \in \mathbb{R}^{n \times n}$ and $\mathbf{\eta} \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the graph) are respectively raised to $\alpha$ and $\beta$, so they could have used also other type of brackets, for example, $()$. It may also be a way of indicating that $[\tau_{ij}]$ and $[\eta_{ij}]$ are $1 \times 1$ matrices or vectors that contain respectively the scalars $\tau_{ij}$ and $\eta_{ij}$, so you are multiplying matrices or vectors (dot product). 

This notation is also used in the paper that introduced the ant colony system (ACS) (and it is probably used in many other papers related to ant colony optimization). See equation 1 of Ant Colony System: A Cooperative Learning Approach to the Traveling Salesman Problem (1997) by Dorigo and Gambardella.