The corporate watchdog is introducing new rules covering the secretive world of dark pools on private exchanges and high-frequency trading.
The Australian Securities and Investments Commission will also set up task forces to investigate both areas.
ASIC said the new rules would help address the risks involved with growth in automated trading and dark pools.
The new rules covering high-frequency trading include volatility controls for extreme price movements and restrictions on automated trading.
Those operating dark pools, away from public investment markets, will also be subject to new requirements covering the validation of trades and extra data about the transactions.
"Developments in trading and market structure domestically and abroad are rapidly shifting the landscape of the Australian market, and we see a trend towards more frequent, smaller trades, away from public markets, with implications for the price formation process," ASIC chairman Greg Medcraft said.
The move by ASIC comes a day after the federal government announced new protections for investors involved in high-frequency trading and dark pools.
The government has also ordered a review of financial licences in Australia, including ones held by investment banks that operate dark pools.